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PREFACE 

In  this  Memorandum,  the  author  describes  the  uses 
and  contributions  of  the  mathematical  theory  known  as 
dynamic  programming  to  certain  problems  In  economics. 
Examples  of  these  eu'e  the  optimal  allocation  of 
resources,  and  multistage  decision  processes  that 
Involve  planning  and  learning  In  the  face  of  uncertainty 
(l.e.,  adaptive  control  processes). 


SUMMARY 


The  fmctlons  of  a  mathematical  theory  In  a 
scientific  field  are  to  furnish  a  systematic  means  of 
formxilatlon  of  classes  of  problans,  to  indicate  various 
techniques  for  their  analysis,  and  to  provide  methods 
for  obtaining  numerical  answers  to  numerical  questions. 

At  one  point  In  the  nineteenth  century,  serious 
doubt  was  expressed  that  problems  In  the  field  of 
economics  could  ever  be  handled  mathematically.  The 
introduction  of  the  digital  computer  changed  the  situa¬ 
tion  drastically.  Nevertheless,  much  remains  to  be 
done,  and  many  new  approaches  must  be  devised,  before 
we  can  consider  ourselves  to  have  a  firm  hold  In  the 
domain  of  mathematical  economics. 

In  this  Memorandum  we  outline  briefly  some  of  the 
principal  contributions  of  the  theory  of  dynamic 
programming  to  the  formulation,  analysis,  and  computa¬ 
tional  treatment  of  economic  processes. 
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DYNAMIC  PROGRAMMING  AND  MATHEMATICAL  ECONOMICS 


1.  INTRODUCTION 

The  fvinctlons  of  a  mathematical  theory  In  a 
scientific  field  are  to  furnish  a  systematic  means  of 
formulation  of  classes  of  pixjblems,  to  Indicate  various 
techniques  for  their  analysis,  and  to  provide  methods 
for  obtaining  numerlciJ.  answers  to  numerical  questions. 

At  one  point  In  the  nineteenth  century,  serious 
doubt  was  expressed  that  problems  In  the  field  of 
economics  could  ever  be  handled  mathematically.  The 
Introduction  of  the  digital  computer  changed  the 
situation  drastically.  Nevertheless,  much  remains  to 
be  done,  and  many  new  approaches  must  be  devised,  before 
we  can  consider  ourselves  to  have  a  flm  hold  In  the 
domain  of  mathematical  economics. 

In  what  follows,  we  outline  briefly  some  of  the 
principal  contributions  of  the  theory  of  dynamic  program¬ 
ming  to  the  formxilatlon,  analysis,  and  computational 
treatment  of  economic  processes.  Further  references, 
examples,  and  detailed  discussion  will  be  fovind  In 

[1,2,3] . 

2 .  ALLOCATION  PROCESSES 

A  fundamental  problem  In  economics  Is  that  of  the 
allocation  of  resources.  Assuming  that  we  possess  the 
basic  utility  functions,  the  mathematical  problem  In 
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many  cases  Is  that  of  maximizing  the  function 
N 

(2*l)  Rjj  ®  ^^11*^12 ^  ***  *^1M^ * 

subject  to  the  constraints 
N 

(2.2)  2  X*  .  <c.,  J  *=1,2,...,M, 

1«1  ^ 

and  >  0. 

Uie  functional— equation  technique  of  dynamic 
pr*ograinmlng  transfoms  this  problem  into  that  of  the 
solution  of  the  functional  equation 

(2.3)  fj^(c^,C2,...,Cj^)  =  m^  tgN(xNi 

R 

where  R  is  determined  by  the  constraints 

(2.4)  ^  ^  ^SfJ  —  ^J*  ^  "  1>2^...,M. 

In  a  few  cases,  this  equation  can  be  treated 
analytically.  In  general,  we  are  Interested  in  computa¬ 
tional  algorithms  vrtilch  are  independent  of  the  analytic 
structure  of  the  functions  g^.  If  the  solution  can  be 
obtained  numerically,  we  have  the  optimal  allocation 
policy  as  a  fvinction  of  the  number  of  activities  and  the 
quantities  of  resources  available. 
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3.  DIRECT  COMPUTATIONAL  APPROACHES 

In  view  of  the  rapid-access  storage  capacity  of  the 
biggest  £ind  fastest  modem  digital  computer,  a  count  of 
required  storage  shows  that  one—  and  two-dimensional 
allocation  processes  are  readily  resolved  in  a  direct 
fashion.  Three-dimensional  allocation  processes  are  on 
the  border  line;  see  [2] ,  Chaps.  1  and  2. 

A  very  simple  analytic  device,  that  of  the  Lagrange 
multiplier,  enables  us  to  increase  the  power  of  the 
method.  In  place  of  the  H  constraints  of  (2.2),  we 
impose  K  constraints, 

J  =  1,2,...,K, 

eind  replace  the  criterion  function  by 

N  M  /  N  \ 

(3.2)  R^(X)  =  2  g  -  2  X  f  2  X.  J. 

^  1=1  ^  J=R+1  J  ^1=1 

As  Is  well  known,  the  economic  Interpretation  of  the  Xj 
Is  that  of  a  price.  The  dynamic— programming  approach 
now  furnishes  the  optimal  allocation  as  a  function  of 
prices  and  resources. 

We  can  now  consider  up  to  four  or  five  different 
kinds  of  resources  without  difficulty. 

4 .  ANALYTIC  NOTES 

If  the  functions  g^^  are  linear,  the  foregoing 
provides  a  dynamic-programming  approach  to  the  general 
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llnear  programming,  an  approach  from  which  a  number  of 
results  can  be  deduced;  see  Appendix  2  of  [2].  These 
methods  can  also  be  applied  to  nonlinear  progranmlng . 

For  an  analytic  approach  to  allocation  processes 
based  upon  the  transfom 

(4.1)  F(x)  «=  max  tf(y)  —  xy] , 

y>0 

see  [  4] ,  [51* 

5.  SOPHISTICATED  COMPUTATIONAL  APPROACHES 

Let  us  now  consider  two  techniques  for  circumventing 
multidimensional  difficulties,  namely,  successive 
approximations  and  polynomial  approximation;  see  [  6l , 

[2I.  As  usual,  the  method  of  successive  approximations 
cannot  guarantee  convergence  to  the  absolute  maximum,  but 
it  can  provide  monotonlclty.  This  is  important,  since 
we  can  use  it  to  test  suiy  proposed  solution  and  thus  to 
improve  anything  which  is  being  done  currently. 

To  Illustrate  an  application  of  the  method,  consider 
the  problem  of  maximizing 
N 

(5.1)  2  g.(x.,y.) 

1-1  ^  ^  ^ 


over  all  and  satisfying  the  constraints 

Xi,  yj^  >  0  and 


c 


2* 


(5.2) 


Let  be  an  initial  approximation. 
Determine  a  new  approximation  by  holding  the  y^^ 
fixed  and  maximizing 


(0) 


N 


(5.3)  2  gj(x.*y^ 

1-1  ^  ^  ^ 


(0) 


) 


over  Xj^  >  0,  2  x^  -  a  ono-dlmenalonal  problem. 

1  /q\ 

This  yields  a  set  (x^'  M.  Keep  this  set  fixed  and 
maximize 

"  (1) 


(5.^)  z 

1-1  ^  ^  ^ 


over  >  0,  2  yji^  «  Cg#  and  so  on.  It  Is  clear  that 

the  maximum  return  Increases  monotonlcally. 

This  technique  can  be  applied  to  allocation 
processes  with  a  considerable  nvnnber  of  resources.  In 
particular,  it  has  been  used  with  success  to  treat  a 
Hitchcock— Koopmans-^antorovlch  scheduling  problem 
Involving  nonlinear  shipping  costs;  see  1 2] . 

Ihe  dimensionality  difficulties  encountered  In  the 
direct  approach  referred  to  In  Sec.  3  were  due  to  the 
storage  of  the  values  of  f]^(c^,C2« . .  •  #0|^)  at  lattice 
points  In  0— space.  Talcing  advantage  of  the  smooth 
b^avlor  of  which  we  can  expect  In  general,  we  can 
store  the  functional  values  by  means  of  a  polynomial 
approximation 

(5.5) 
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where  the  Sj(c)  are  polynomials.  If  the  Sj(c)  are 
fixed  in  advance,  storage  of  fjj(c)  reqxilres  only 
storage  of  the  coefficients  By  means  of  a  few 

hundred,  or  a  few  thousand,  coefficients,  we  can  obtedn 
excellent  approximations.  In  this  way,  we  trade  time, 
vrtilch  we  possess,  for  space,  vdiich  we  do  not.  See  [2] 
and  [6]  for  detsiiled  discussion  and  applications. 

6.  MULTISTAGE  DECISION  PROCESSES 

In  the  main,  dynamic  programming  is  most  useful  in 
the  study  of  multistage  processes  requiring  seciuences  of 
decisions.  In  the  parlance  of  control  theory,  it  is 
most  powerful  in  the  treatment  of  feedback  control 
processes.  Processes  of  this  type  arise  frequently  in 
economics,  multistage  investment,  purchasing,  inspection, 
and  repair  of  equipment,  multistage  production  processes, 
Inventory  theory;  see  ll],  [2].  Many  of  these  processes 
are  Maricovlan  decision  processes,  such  as  those  arising 
in  Inventory  and  repair  theory;  see  the  various  volmes 
edited  by  Arrow,  Karlin,  and  Scarf  [7]  for  numerous 
examples  of  the  analysis  of  dynamic  programming  processes. 
Many  further  references  will  be  fovind  in  [1,2,3]. 

The  basic  functional  equation  takes  the  form 

(6.1)  fj^(p)  -  max  [g{p,q)  +  2  Wj^fj^^(Tj^(p,q) )]  . 

An  advantage  of  this  formulation  is  that  deterministic 
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and  stochastic  processes  are  treated  In  the  same 
conceptual,  analytic,  gmd  computational  fashion. 

The  computational  techniques  described  above  can 
all  be  applied  to  this  ec[uatlon,  as  well  as  that  of 
"approximation  in  policy  space."  (See  [1,2,3].) 

7.  ADAPTIVE  PROCESSES 

In  many  processes  involving  planning  in  the  face  of 
uncertainty,  the  classical  techniques  are  not  Immediately 
applicable  due  to  the  absence  of  Infomatlon  which  we 
usueilly  tcike  for  granted,  e.g.,  distributions  of  random 
variables,  cause  and  effect,  and  so  on;  see  [3]* 

In  these  cases,  it  is  necessary  to  make  decisions 
and  to  learn  about  the  process  at  the  same  time.  The 
theory  of  dynamic  programming  provides  a  unlfom  approach 
to  decision  processes  of  this  type,  wlilch  we  call 
adaptive ♦  Detailed  discussion  and  examples  are 
presented  in  [  3]  •  The  simple  and  Intviltive  Principle  of 
Optimality  provides  the  conceptual  framework  upon  vdilch 
analytic  approaches  can  be  based.  This  principle  can  be 
stated  as  follows: 

Principle  of  Optimality.  An  optimal  policy  has  the 
property  that  vdiatever  the  initial  state  and  initial 
decision  are,  the  remaining  decisions  must  constitute  an 
optimal  policy  with  regard  to  the  state  resulting  from 
the  first  decision. 
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As  before,  con5)utatlonal  difficulties  arise  because 
of  the  presence  of  functions  of  many  variables  and. 
Indeed,  functions  of  functions. 

8 .  SIMULATION  PROCESSES 

One  of  the  advantaces  of  the  emphasis  on  the  concept 
of  policy  In  dynamic  programming  Is  that  multistage 
decision  processes  with  no  criterion  for  optimal  outcome 
can  still  be  fruitfully  studied.  Combining  digital 
computers  with  mathematical  fomulatlon,  we  can  employ 
simulation,  or  "gaming"  techniques;  see  [8] . 

9.  CONCLUSION 

At  the  present  time,  we  possess  a  number  of 
powerful  mathematical  techniques  for  the  analytic  and 
con5)utatlon2Ll  solution  of  classes  of  problems  In  the 
field  of  mathematical  economics.  What  Is  now  needed  Is 
a  systematic  exploitation  of  these  methods  to  provide  a 
backlog  of  solved  problems  which  will  gtilde  our 
subsequent  research. 
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